Conversation
…ith a MESSAGE prefix match
eligotts
changed the title
| prev_turn_completion_ids = prev_turn_tokens["completion_ids"] | ||
| prev_turn_ids = prev_turn_prompt_ids + prev_turn_completion_ids | ||
|
|
||
| def normalize_for_comparison(value: Any) -> Any: |
There was a problem hiding this comment.
should we make this a general message_util? seems useful in other places too? also, vaguely remember we have a similar util to this alr but might be wrong
|
|
||
| return 0 | ||
|
|
||
| # we add suffix_ids to prev_turn_ids. suffix_ids are tokens that are added |
There was a problem hiding this comment.
i know unrelated to this pr but can i think we might be able to remove the suffix part since we tokenize the env_response_ids = full_ids[len(prev_turn_ids) :] and not tokenize env response ids in isolation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| best_step_tokens = step_tokens | ||
| if best_prefix_len == len(normalized_prompt_messages): | ||
| break | ||
|
|
There was a problem hiding this comment.
Per-turn backward scan may be expensive
Medium Severity
get_prompt_ids() now walks backward over the entire state["trajectory"] and calls to_native_prompt() per step until it finds the best prefix, which can add significant overhead on long trajectories and slow every generation turn.
| if best_step_tokens is None: | ||
| return None | ||
| return best_step_tokens["prompt_ids"] + best_step_tokens["completion_ids"] | ||
|
|
There was a problem hiding this comment.
Prefix match can miss equivalent messages
Medium Severity
get_prompt_ids’s new message-level prefix matcher compares normalized message objects for strict equality, which can differ across representations (e.g., to_native_prompt emitting {"content": None} while incoming prompt_messages omits content, or other default/extra fields). This can produce false “no prefix match”, disabling the token route unexpectedly.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
|
|
||
| prev_turn_ids = await find_largest_prefix_match_tokens() | ||
| if prev_turn_ids is None: | ||
| return None |
There was a problem hiding this comment.
Prefix match ignores tool-dependent tokenization
Medium Severity
find_largest_prefix_match_tokens() selects a trajectory step using only a message-level prefix comparison, but the stitched prev_turn_ids are later combined with full_ids produced by tokenize(..., tools=oai_tools). If the effective tool set differs from when the matched step’s tokens were produced, prev_turn_ids may not align with full_ids, yielding incorrect env_response_ids and an invalid prompt for /chat/completions/tokens.
2 participants
Description
Best effort tito. instead of assuming extension and looking back at strictly last trajectory step, we walk backward until we find a MESSAGES level prefix hit
tested on both wiki-search and eligottlieb/poker-multiagent, which was giving the loud failure previously with TITO as it does explicit rewriting of history (like context folding)
Type of Change
Testing
uv run pytestlocally.Checklist
Additional Notes
Note
Medium Risk
Changes how token-stitching selects prior-turn tokens and adds a new fallback path; mistakes could cause silent behavior changes (more MITO calls) or incorrect token prefixes in production.
Overview
Makes token-stitching (TITO) best-effort:
get_prompt_idsnow scans the trajectory backwards to find the largest message-level prefix match (normalizing message structures) instead of assuming the last step matches, and returnsNonewhen no match exists.Updates
get_native_responseto fall back to the standard/chat/completionspath whenget_prompt_idsreturnsNone, avoiding broken/chat/completions/tokensrequests on history rewrites/context folding. Adds async tests covering largest-prefix selection, no-prefix behavior, and ensuring the correct route is used depending on whether prompt token IDs are available.Written by Cursor Bugbot for commit 6ae6aff. This will update automatically on new commits. Configure here.